Evaluation of data processing pipelines on real-world electronic health records data for the purpose of measuring patient similarity

Background The ever-growing size, breadth, and availability of patient data allows for a wide variety of clinical features to serve as inputs for phenotype discovery using cluster analysis. Data of mixed types in particular are not straightforward to combine into a single feature vector, and techniques used to address this can be biased towards certain data types in ways that are not immediately obvious or intended. In this context, the process of constructing clinically meaningful patient representations from complex datasets has not been systematically evaluated. Aims Our aim was to a) outline and b) implement an analytical framework to evaluate distinct methods of constructing patient representations from routine electronic health record data for the purpose of measuring patient similarity. We applied the analysis on a patient cohort diagnosed with chronic obstructive pulmonary disease. Methods Using data from the CALIBER data resource, we extracted clinically relevant features for a cohort of patients diagnosed with chronic obstructive pulmonary disease. We used four different data processing pipelines to construct lower dimensional patient representations from which we calculated patient similarity scores. We described the resulting representations, ranked the influence of each individual feature on patient similarity and evaluated the effect of different pipelines on clustering outcomes. Experts evaluated the resulting representations by rating the clinical relevance of similar patient suggestions with regard to a reference patient. Results Each of the four pipelines resulted in similarity scores primarily driven by a unique set of features. It was demonstrated that data transformations according to each pipeline prior to clustering can result in a variation of clustering results of over 40%. The most appropriate pipeline was selected on the basis of feature ranking and clinical expertise. There was moderate agreement between clinicians as measured by Cohen’s kappa coefficient. Conclusions Data transformation has downstream and unforeseen consequences in cluster analysis. Rather than viewing this process as a black box, we have shown ways to quantitatively and qualitatively evaluate and select the appropriate preprocessing pipeline.


Journal Requirements:
When submitting your revision, we need you to address these additional requirements.
1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations. pdf We have revised the manuscript formatting style according to the guides.
2. Please provide additional details regarding participant consent. In the ethics statement in the Methods and online submission information, please ensure that you have specified what type you obtained (for instance, written or verbal, and if verbal, how it was documented and witnessed). If your study included minors, state whether you obtained consent from parents or guardians. If the need for consent was waived by the ethics committee, please include this information We have provided further clarification on the data sources subsection and the ethics statement.
"In all studies using CPRD data, consent is not given on an individual patient level. Selected practices…" [Note: HTML markup is below. Please do not edit.] Reviewers' comments:

Reviewer's Responses to Questions
Comments to the Author 1. Is the manuscript technically sound, and do the data support the conclusions?
The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data-e.g. participant privacy or use of data from a third partythose must be specified.

Reviewer #1: Yes
Reviewer #2: No 4. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Review Comments to the Author
Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: The aim of this paper was to outline and implement an analytical framework to evaluate four different data processing methods for constructing patient representations from EHR data for measuring patient similarity. The authors also evaluated the influence of features on patient similarity and the effect of different data processing methods on Euclidean distance-based cluster analysis.
This manuscript could be improved by addressing the following major and minor concerns.
We thank Reviewer #1 for their comments and constructive feedback.
Major concerns: 1. The term 'semi-supervised evaluation' was emphasized in the title of the manuscript. However, it was unclear how the semi-supervised evaluation was carried out. Typically, in a semi-supervised method, there was a small proportion of samples with supervised labels, and a large proportion of samples without supervised labels. What were the labels in the current study? How many samples had labels? How were these labels used in the evaluation?
Thank you for raising this. The term "semi-supervised" has been used loosely to refer to the expert clinicians' involvement in the selection of the best method, rather than the use of labels for a portion of the dataset. We propose a corrected title omitting the term "semisupervised" and have corrected the use of the term where it appears in the main text: "Evaluation of data processing pipelines on real-world electronic health records data for the purpose of measuring patient similarity." 2. The proposed data processing pipelines used PCA, MCA, and autoencoders for patient representation, which are less used in the era of deep learning. Therefore, the value of this manuscript is greatly discounted. I suggest that much more advanced representation methods should be used or reproduced and further evaluated.
Thank you for this comment. PCA and MCA are extremely popular and powerful methods that are being heavily utilized in the field of disease phenotyping, as evidenced by the results of our literature search. As a result, investigating their ability to represent complex data such as medical history information, is of interest and is valuable. We selected the autoencoder as a representative deep learning method. We agree that examining other more advanced representation methods would be interesting, but this would have to be in a new manuscript as it is outside the scope of our current research manuscript.
We have revised the manuscript to include a sentence in the conclusion on page 17 accordingly: "It is feasible and of potential interest to apply this analysis to more advanced, deep-learning approaches." 3. Related to the previous comment, the references cited in the manuscript were too old. Up to 20 out of the 33 references were published 10 years ago, even in the 1960s. The authors only reviewed literature dealing with mixed-data types in COPD subtyping studies. I don't think that's enough. My suggestion is to widen the scope of the search without limiting the diseases studied, because the proposed pipelines were not specific to COPD.
There are three points to this comment that we have addressed as follows: 1. References as old as the 1960s: These are references pertaining to methodology, and we decided to select and cite early foundational references to these concepts, such as the "curse of dimensionality" (Belman, 1961) and statistical tools, such as Cohen's kappa coefficient (Fleiss et al. 1969, Landis and Koch 1977) on the basis that they are in use currently as originally introduced.
2. Scope of the literature survey: We chose COPD as an example of a typical "umbrella term" disease entity that researchers have tried to sub-phenotype using cluster analysis. We, therefore, restricted the literature search to COPD papers as it was not our intention to systematically review every possible application of cluster analysis. 4. The term "learned representation" appeared many times throughout the text. As far as I know, PCA and MCA were not a kind of "learning" methods. Representations using either method were obtained only by computation, not by learning or training.

Inclusion of more recent studies in
Thank you for pointing this out. We have revised the manuscript to ensure terminology is consistent with the methods used either simply as "representation" in the case of MCA/PCA methods or "learned representation" for the autoencoder.
5. What is the relationship between the evaluation of patient representation methods and the identification of important features? Had these important features been clinically validated?

The features used as input to the data processing pipelines have all been selected by respiratory consultant clinicians on the basis of their clinical relevance the diseases. We have included detailed information on feature selection and definitions in a new subheading under
Methods "Feature selection" on page 4.

"Feature Selection
The following features were selected as input to the analysis on the basis of their clinical relevance to COPD as described previously ( We calculate the relative variance metric as a measure of how much each of the original features contributes to the similarity matrix. This is a metric of "algorithmically assigned" feature importance rather than a metric of the clinical importance of the feature. In terms of method evaluation, the calculation of this metric allows the user to understand the handling of each feature and its relative importance in the construction of the similarity matrix. In practice, it can allow the user to spot when feature encoding results in the disproportionate and undesirable weighting of a certain type of feature, as can happen for example with categorical over numerical features. Table 4) in the revised manuscript which summarises the evaluation of each pipeline according to feature importance, cluster tendency (as per comment 7, see below) and clinical expert ranking on page 14.

We used a variety of metrics to evaluate the four data representations. Table 4 below summarises the results. The selection of relevant metrics will ultimately depend on the use case. For example, while it might be desirable to compute feature importance for representations fed to a supervised learning model, the Hopkins index is only relevant to cluster analysis."
Minor comments 6. Table 1 is not referred to in the main text.
Thank you for pointing out this omission. We have added a new subsection under Results "Cohort characteristics" that references Table 1 on page 9.

"Cohort characteristics
The study comprised 30,961 patients from 393 primary care practices. The characteristics, which include all features used in the analysis of the overall cohort, and the training and testing dataset partition are shown in Table 1." 7. More indices, such as Hopkins statistics, Silhouette index, and Davies-Bouldin index, should be used to evaluate the clustering solutions.
We used the percentage of patients that cluster together as an indicator of clustering similarity between each set of representations. The aim of this comparison was to illustrate the difference pre-processing of data can make to the clustering results. While the Silhouette and Davies-Bouldin indices are used to (internally) evaluate clustering solutions, we would suggest this is a separate research question to the one our study is aiming to answer, which is the choice of pre-processing the data prior to clustering.
The Hopkins index is however a valuable index to compute, as it is a measure of cluster tendency of the dataset, and can thus be applied to each representation. We thank the reviewer for this recommendation. We have calculated the Hopkins index and included it in our results and discussion as an additional potentially useful metric which can help guide the choice of representation intended for cluster analysis. We have provided this information under Methods: Clinical evaluation of patient similarity on page 8:

"…The reference patient and case matches were presented to two clinical experts (Consultants in Respiratory medicine)…"
Reviewer #2: In their work, the authors investigate how data with different types of features (numeric, categorical, ordinal) can be processed to assess similarity between data instances without biasing similarity to the feature type. They do so within the setting of patients with COPD, and assess clinical agreement with the created patient clusters.
We thank Dr Zwanenburg for their comments and constructive feedback.
1. The introduction lacks a clearly stated objective, aim or hypothesis. The introduction seems to steer the reader towards an investigation of methods that can deal with a large feature space of mixed feature types in a bias-free manner. However, I don't think this question can be confidently answered using the study setup presented in this work. The authors should edit the introduction to be more specific regarding the research question that their study attempts to answer.
We thank the reviewer for this important observation. We have included a "study aims" subsection under "Introduction" on page 3:

"Study aims
The primary aim of this work was to describe and implement an approach to evaluating data representations and subsequent impact on data-point similarity resulting from a variety of data processing pipelines. This evaluation includes: 1. The investigation of assigned feature importance in calculating data point similarity, including the relative contributions of numeric and categorical features 2. The clinical evaluation of resulting similarity relationships by expert clinician raters 3. The evaluation of cluster tendency of the resulting representations The above metrics can be individually considered in order to select an appropriate processing algorithm for the desired application, in this case cluster analysis, hence the inclusion of cluster tendency as an additional metric.
The secondary aim of this work was to demonstrate that decisions on data pre-processing have downstream effects on clustering results." 2. The authors focus heavily on representing patients in a lower-dimensional space. This has two potential drawbacks: a. Depending on the final aim of the data processing pipeline (e.g. supervised learning), this may limit explainability. b. Not all features are equally valuable for assessing similarity between patients. In the dataset employed by the authors, all features are at least plausibly meaningful. However with more data becoming available, features may also insert noise, and create irrelevant dissimilarity. I do not expect from the authors that they investigate other methods for this publication, but they might make note of such issues for further work.
We thank the reviewer for raising these important observations. We have added relevant sections under study limitations page 17: "Representing patients in a lower dimensional space may limit the explainability of results, while the increase in data availability can introduce irrelevant features which can add noise to the calculated similarity metrics." 3. Though the algorithms are in a sense agnostic to what features are clinically relevant for finding patients, the clinicians who performed the assessment likely do have their preference, i.e. for grouping smoking and non-smoking patients. Thus a feasible alternative to the representation-based methods presented by the authors for clustering similar patients is to use expert consensus on important clinical features and compute Gower's distance between patients for sampling.
The reviewer makes an interesting point, however, we chose cluster analysis since it's a widely used method. In clinical applications of cluster analysis, the selection of features is generally agreed upon by clinicians, as is the case in our study.
We agree that a different metric could also be explored (and indeed although we use it in this study, our method is not limited to the Euclidean distance), however, the proposed Gower's distance would only be appropriate for categorical features, and therefore the issue of handling mixed data types remains.

Which loss function was used to train auto-encoders?
The loss function used was the mean squared error. This has been updated in the text under the Methods: Autoencoders subheading on page 6: "…..the results based on a loss function (mean squared error) calculated on both the training and test dataset." 5. Computing the relative variability metric requires some steps that are not properly explained: a. The concept of pairwise agreement is used, but I did not understand how agreement is assessed. b. A reference patient is required, but it is not clear how this patient is selected.
Thank you for this comment, we agree it is important to clarify these points in the text. a. We have corrected the text and specified pairwise disagreement, instead of agreement, as we are calculating a distance metric. Pairwise disagreement is a way to quantify dissimilarity in categorical features. Two data points either share the same value of a categorical feature, therefore are in agreement, or they have different values and are in disagreement. To quantify the average disagreement between a patient and a sample of their neighbouring patients for a categorical feature we used the proportion of disagreement. b. We have removed the word "reference", as it may confuse with the reference patient mentioned in the clinical evaluation section. Instead, we specify that the neighbouringpatient feature similarity is calculated for every patient.
6. To what degree is the cluster analysis shown in 3.5 sensitive to the composition of the patient dataset? I.e. if the cluster analysis is repeated multiple times using the same method but with subsets (e.g. bootstraps) of the data, how often do patients cluster together in the same cluster? Currently it is unclear if the presented values in table 4 are due to inherent differences between representation methods, or are close to the upper limit of what may be expected given the dataset.
Thank you for making this very important point and suggestion. We re-sampled 10% of the dataset 20 times for each method and each number of clusters k, and repeated the cluster analysis. We reported the average percentage of agreement, (percentage pairs of patients who remain clustered together in each sample) in the diagonals of Table 3.
7. Please be advised that though the authors may be prohibited from sharing the raw data (even though they claim these are fully anonymised), PLOS ONE does require that the data underlying the presented results should be published, e.g. those underlying Figure 6. See https://journals.plos.org/plosone/s/materials-software-and-code-sharing for more information.
We are unfortunately unable to share the raw data pertaining to the patient records used, and access is only possible for approved researchers, following CPRD's Research Data Governance Process, as described in the following guide: https://cprd.com/safeguardingpatient-data ; this is in line with the data providers governance framework we operate under and while data are indeed anonymized, the risk of identifiability still remains.
We have provided the raw ratings data underlying Figure 6. This is now included in the Supporting Information file.